A New Torus-like Mapping for Parallel Sparse Matrix Factorization

In Cle93] we describe a new mapping of sparse matrices to the processors of a distributed memory parallel computer, called the sparse torus wrap mapping (STWM), designed to reduce the volume of interprocessor communication during the Cholesky factorization A = LL T. The mapping combines the advantages of the so-called dense torus wrap mapping (DTWM) Ash91b] developed for dense matrix factorizations with those of domain decomposition-type column oriented mappings GLN89] developed to take advantage of the structure of sparse matrices. The paper Cle93] was mainly concerned with a classical sparse matrix model problem: a k k grid with a nine-point stencil with n = k 2 > p unknowns ordered by nested dissection Geo73]. The elimination tree GL81] corresponding to the model problem is extremely well balanced and thus not indicative of typical problems. In this paper we extend the mapping to arbitrarily unbalanced elimination trees by use of a proportional mapping scheme, and thus show the utility of the mapping for general problems. We present a simple algorithm that automatically performs the assignment of elements to processors' memories. The main theoretical quantity of interest in this work is the total communication volume (the amount of oating-point numbers that must be communicated from one processor's physical memory to another processor). If the communication volume for an algorithm is greater in an order-of-magnitude sense than total oating point operations, then that algorithm is not scalable GMB88]. It was shown theoretically Sch92] that no mapping of the columns would make a column oriented algorithm for the model problem scalable. For the model problem, GHLN88] gives a straightforward fan-out algorithm whose resultant communication volume is O(pk 2 log 2 k). By changing the mapping of the columns to a domain decomposition-type subtree to subset mapping, GLN89] were able to modify their fan-out algorithm to require O(pk 2) communication volume in O(pk log k) messages. For a restricted family of column oriented algorithms, they showed that their communication volume was optimal. For the case p < k, HZ91] reduced the number of messages to O(pk log p) but retained the same communication volume. MR92] further reduced the number of messages to O(pk), but reduced only the constant for the communication volume. Ash91a] argued that by violating the assumption of MR92] there existed a column oriented algorithm that would lower the communication volume to O(p 1=2 k 2) but at the cost of increasing the …